\chapter{Installation and execution of the code}

\section{Stand-alone mode}

\subsection{CVS installation}
Full command for AMPS checkout:\\\\
{\tt cvs -d username@herot.engin.umich.edu:/CVS/FRAMEWORK checkout AMPS}
\subsection{SPICE installation}
Go to the following URL to download the newest version of SPICE (cspice)\\\\ \url{http://naif.jpl.nasa.gov/naif/toolkit_C.html}\\\\
choose the proper version and download these two files
\begin{itemize}
 \item cspice.tar.Z
 \item importCSpice.csh
\end{itemize}
Copy these two files into a folder where you want to have you SPICE installation, then\\\\
{\tt /bin/csh importCSPice.csh}\\\\
for installation



\subsection {Installation command}
\begin{enumerate}
\item Add current directory '.' to your PATH, i.e. put  {\tt PATH=\$PATH:.} in your\\
.bashrc file (.bash\_profile for OSX)
\item Make sure that CVS is installed on your computer. 
\item Then go to a folder you wish to install AMPS in via terminal. Then type: \\
{\tt CVS checkout AMPS }\\
AMPS is being set up into a subfolder called AMPS.
\item Now move into the AMPS folder:\\
 {\tt cd AMPS}
\item Now choose an application you wish to use and install the code regarding your interest.
In case you want to do a simulation of the Moon, type the following into the command line:\\
{\tt Config.pl -install -application=moon}\\
You may choose one of the following existing applications:\\ 
{\tt  bullet}, {\tt CG}, {\tt CouplerTest}, {\tt Enceladus}, {\tt Europa}, {\tt Hartley2},
{\tt Hartley2RotationBody}, {\tt Interface}, {\tt Mercury}, {\tt Moon}, {\tt Rosetta}, {\tt Shock}\\
ToDo: SHORT DESCRIPTION OF ALL THESE PRE-DEFINED CASES
\item Set the path for SPICE and for SPICE Kernels by typing the following two lines:\\
{\tt Config.pl -spice-path=/Users/Username/SPICE/cspice }\\
{\tt Config.pl -spice-kernels=/Users/Username/SPICE/Kernels }\\\\
If you do not need any SPICE functions for you simulation it can be turned off by\\\\
{\tt Config.pl -spice-kernels=nospice}
\end{enumerate}
Now the code is ready to compile. Type {\tt make} into the terminal and compile the code. An executable file called ''{\tt amps}'' appears in the AMPS folder. You may run the code now with the executable or submit it via a job-file on a supercomputer. It takes 6-8 hours to run on a single core. The output files are generated under the AMPS folder and data outputs are in the PT/plots folder.

\subsection{Arguments for {\tt Config.pl}}
When using the Config.pl for setting up the code we give values to its arguments by using the {\tt =} sign, for example:\\
{\tt Config.pl -install -application=moon}\\
passes the value {\tt moon} for the {\tt application} argument.\\
The possible arguments are:
\begin{itemize}
\item {\tt Config.pl -application } when setting up the code before compiling it we can choose the object of our interest: {\tt  bullet}, {\tt CG }, {\tt CouplerTest }, {\tt Enceladus }, {\tt Europa }, {\tt Hartley2 }, {\tt Hartley2RotationBody }, {\tt Interface }, {\tt Mercury }, {\tt Moon }, {\tt Rosetta }, {\tt Shock}\\
\item {\tt Config.pl -help } self-explanatory.\\
\item {\tt Config.pl -ices-path } in case we use the code in stand-alone mode, we have to use a pair of input files generaded by BATSRUS: icesCellCenterCoordinates.MHD.dat and ices.data.dat. We have to set the path for these files with this argument. \\
\item {\tt Config.pl -show} shows the current settings of Config.pl and AMPS, showing the paths and application we currently use.\\
\item {\tt Config.pl -spice-kernels } sets the path for SPICE Kernels.\\
\item {\tt Config.pl -spice-path } sets the path for SPICE.\\
\end{itemize}



\section {Component of SWMF}
\subsection {Passing arguments from SWMF's Config.pl}
\subsection {Reading SWMF's PARAM.in file}
\subsection {Debugging}


\subsection{Execution of AMPS in hybrid (MPI+OpenMP) mode as a component of the SWMF}

\subsubsection{Running of AMPS on the same nodes with BATSRUS}

{\bf The SWMF LAYOUT.in}
\\
{\it This is the layout for running OH and PT on ivy nodes.\\
\\
Name First Last Stride\\
======================\\
\#COMPONENTMAP\\
PT       0 9999    20   ! PT runs on all PE-s\\
OH       0 9999    1   ! OH runs on all PE-s\\
\#END\\
}



{\bf The job script on Pleiades:}
\\
{\it \#PBS -S /bin/csh\\
\#PBS -N SWMF\\
\#PBS -l select=4:ncpus=20:model=ivy\\
\#PBS -l walltime=01:00:00\\
\#PBS -q devel\\
\#PBS -j oe\\
\#PBS -W group\_list=s0799\\
\#PBS -m e\\
module load comp-intel/2015.3.187\\
module load mpi-sgi/mpt\\
cd \$PBS\_O\_WORKDIR\\
\\
setenv OMP\_NUM\_THREADS 20\\
\\
setenv MPI\_MSGS\_PER\_HOST 100000\\
setenv MPI\_MSGS\_PER\_PROC 100000\\
setenv MPI\_MSGS\_MAX 100000\\
mpiexec ./SWMF.exe > runlog\_\'date +%y%m%d%H%M\'\\
\\
exit\\
\\
if(! -f SWMF.SUCCESS)exit\\
}






\subsubsection{Running of AMPS on nodes separate from those used by BATSRUS}


{\bf The layout for running OH and PT:}
\\
{\it Name First Last Stride\\
======================\\
\#COMPONENTMAP\\
PT       0 1    1   ! PT runs on all PE-s\\
OH       2 9999    1   ! OH runs on all PE-s\\
\#END }


{\bf The job script for Pleiades:}
\\
{\it 
\#PBS -S /bin/csh\\
\#PBS -N SWMF\\
\#PBS -l select=2:mpiprocs=1:ompthreads=40:model=ivy+2:mpiprocs=20:model=ivy\\
\#PBS -l walltime=01:00:00\\
\#PBS -q devel\\
\#PBS -j oe\\
\#PBS -W group\_list=s0799\\
\#PBS -m e\\
module load comp-intel/2015.3.187\\
module load mpi-sgi/mpt\\
cd \\\$PBS\_O\_WORKDIR\\
\\
setenv MPI\_DSM\_DISTRIBUTE 0\\
\\
setenv MPI\_MSGS\_PER\_HOST 100000\\
setenv MPI\_MSGS\_PER\_PROC 100000\\
setenv MPI\_MSGS\_MAX 100000\\
mpiexec ./SWMF.exe > runlog\_\'date +%y%m%d%H%M\' \\
\\
exit\\
\\
if(! -f SWMF.SUCCESS)exit\\
}

\section {Restart Files}
AMPS allows users to save particle data at a customed frequency and also to restart from saved particle data. One can also choose overwrite existing restart files (``overwrite'') or create new files (``newfile'') when saving the restart files. \\
\begin{verbatim}
Examples in input files:
(save restart files)
 SaveParticleRestartFile=on \\  !on,off
 file=ParticleData.restart \\
 SaveMode=overwrite  \\ !overwrite, newfile
 IterationInterval=20

(read restart files)
RecoverParticleData=on    \\ !on, off
file=ParticleData.restart
 
\end{verbatim}
Users can also choose to save and read additional data in the restart files. In order to do this, users need to define the functions to save and read the customized additional information in the restart data. The argument of the functions are pointers to the restart file. Then the function PIC::Restart::SetUserAdditionalRestartData() is called to  pass the user defined functions to core modules. \\
Below is one example of applying the functions.
\begin{verbatim} 
...

void saveRestartData(FILE* fname){
// Only the root processor can write.
  if (PIC::Mesh::mesh->ThisThread==0) {
    fwrite(&iterNumber, sizeof(long int),1, fname);
    std::cout<<"save iter number="<<iterNumber<<std::endl;
  }
}

void readRestartData(FILE* fname){
  fread(&iterNumber, sizeof(long int),1, fname);
  std::cout<<"read iter number="<<iterNumber<<std::endl;
}

...

int main(){
...
 PIC::Restart::SetUserAdditionalRestartData(&readRestartData,&saveRestartData);         
// should be put before the iteration loop.
...
  for (long int niter=0;niter<nTotalIterations;niter++) {

...
  }







}

\end{verbatim}
